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Executive  Summary 


The  goals  of  this  project  are  to  develop,  verify,  validate,  assess,  and  transition  practical,  relevant, 
and  significant  methods,  procedures,  and  tools  that  provide  objective  and  quantified  indicators 
of  acquisition  program  cost,  schedule,  and  performance  risk.  The  methods,  procedures  and  tools 
are  intended  to  complement,  not  replace,  current  risk  management  practices  by  providing  early 
warning  of  risk  exposure,  i.e.,  of  conditions  that  tend  to  increase  the  likelihood  and/or  severity 
of  adverse  acquisition  outcomes. 

The  primary  products  are  (1)  a  set  risk  leading  indicators,  i.e.,  objective  metrics  calculated  from 
standard  program  documents  and  contract  data  reporting  items  that  provide  early  warning  of 
areas  of  risk  exposure,  and  (2)  methods  to  calibrate  and  baseline  the  relationships  between  the 
risk  leading  indicators  and  program  outcomes.  Secondary  products  are  (3)  recommendations  for 
Request  for  Proposal  (RFP)  language  to  improve  visibility  and  transparency  of  risk  exposure 
during  Engineering  and  Manufacturing  Development  (EMD),  and  (4)  recommendations  for 
collecting  and  sharing  data  to  calibrate/baseline  risk  estimating  relationships  across  programs. 
The  risk  leading  indicators  point  to  root  causes,  and  indicate  the  technical  areas  of  risk  in  the 
program. 

The  risk  leading  indicators  address  (1)  evidence  of  risk  exposure  prior  to  EMD  award,  and  (2) 
evidence  of  risk  exposure  during  EMD  execution.  The  risk  leading  indicators  are  intended  for  use 
by  Government  analysts  and  risk  managers,  and  by  EMD  contractors  in  their  risk  management 
program.  The  risk  leading  indicators  are  being  co-developed  working  with  end-users,  and  are 
being  validated  and  assessed  by  application  to  a  major  defense  acquisition  program,  in  concert 
with  the  program's  Risk  Manager.  This  report  describes  progress  and  results  during  2014, 
focused  on  developing  and  verifying  risk  leading  indicators,  and  plans  for  2015,  focusing  on  pilot 
application  and  evaluation  in  cooperation  with  a  major  defense  acquisition  program. 

This  research  develops  the  concepts  of  risk  exposure,  risk  leading  indicators,  and  risk  estimating 
relationships.  The  goal  is  to  produce  methods,  procedures,  and  tools  for  risk  early  warning, 
source  and  cause  diagnosis,  to  facilitate  pre-emptive  risk  identification  and  mitigation.  This 
approach  complements  traditional  technical,  cost  and  schedule  risk  management  methods  and 
stovepipes. 

Risk  exposure  refers  to  conditions  that  amplify  the  likelihood  and/or  consequences  of 
unanticipated  complications,  technical  difficulties  and  delays.  Exposure  to  risk  increases  the  risk 
of  adverse  acquisition  outcomes.  Exposure  to  risk  is  created  by  overly  optimistic  goals  that  lead 
to  inadequate  margins  for  error,  aggressive  concurrent  schedules  counting  on  "things  to  come 
together  at  the  end,"  deferred  or  limited  testing,  coordination  shortcuts,  adopting  novel 
integration  processes  and  promising  but  immature  technologies.  Unstable,  inconsistent, 
incompletely  resolved,  and  highly  interdependent  program  plans  and  system  engineering 
documents  both  indicate  and  create  risk  exposure.  Lagging  and  uneven  technical  progress 
relative  to  the  plan  is  a  further  indicator  of  risk  exposure. 

Risk  leading  indicators  are  objective  metrics  of  risk  exposure  calculated  from  evidence  in 
standard  program  management  and  system  development  reports  and  data.  Risk  leading 
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indicators  provide  information  to  diagnose  the  areas  and  types  of  risk  exposure,  causes  and 
effects.  Risk  leading  indicators  point  back  to  root  causes  and  forward  to  effects.  There  are  many 
different  potential  sources  of  risk  exposure,  requiring  different  sources  of  evidence  and 
calculations.  The  project  is  developing  risk  leading  indicators  that  are  (a)  practical  (i.e.,  can  be 
computed  from  "standard"  program  management  and  system  engineering  documentation)  and 
(b)  relevant  and  significant  (i.e.,  explain  a  large  proportion  and  amount  of  variance  between 
budgeted  and  actual  outcome). 

Risk  estimating  relationships  are  statistical,  evidence-based,  quantitative  relationships  to 
forecast  the  magnitude  and  uncertainty  in  program  cost  and  schedule  overrun,  and  performance 
shortfall  from  the  risk  leading  indicators.  Risk  estimating  relationships  address  the  bias  and 
uncertainty  between  actual  and  planned  time,  cost  and  performance  of  program  activities, 
related  to  program  maturation  by  technical  area.  Risk  estimating  relationships  are  similarto  cost 
estimating  relationships  in  traditional  "analogy"  models.  Calibration  must  balance  the  sample 
set  size  and  uniformity.  Risk  estimating  relationships  address  both  the  bias  and  dispersion  in  time 
and  cost  estimation  error,  which  makes  calibration  more  sensitive  to  differences  between 
programs  than  simple  time  and  cost  estimating  relationships.  Rather  than  attempt  to  forecast 
from  disparate  programs,  we  have  developed  methods  to  accumulate  evidence  from  the 
individual  program-of-interest  to  calibrate  estimation  bias  and  uncertainty.  Risk  of  overrun  is  a 
function  of  the  estimation  bias  and  dispersion  (uncertainty). 

The  approach  is  practical  and  relevant.  It  is  tightly  linked  to  standard  deliverable  data.  It  builds 
on  the  program  risk  evaluation  frameworks  and  criteria  set  out,  in  formal  documentation,  for 
proposal  and  contact  execution  evaluation  as  determined  by  the  PMO.  It  builds  on  prior 
empirical  analyses  of  system  development  leading  indicators,  program  risk  leading  indicators, 
and  root  causes  and  causal  mechanisms  of  adverse  acquisition  outcomes. 

This  approach  to  acquisition  program  risk  early  warning  complements  traditional  risk 
management  practice  and  framework.  The  traditional  risk  management  framework  views  a  risk 
as  a  specific,  identified  potential  future  event,  which  has  a  known  or  estimated  likelihood  of 
occurrence  and  time/cost  consequence  if  it  occurs.  Risk  exposure  addresses  evidence  of  program 
conditions  that  elevate  the  potential  for  time  and  cost  overruns,  and  technical  performance 
deficiencies.  Risk  exposure  analysis  points  to  problem  areas,  for  possible  mitigation.  Risk 
exposure  analysis  is  not  limited  to  "critical  technology  elements"  but  considers  the  entire 
program. 

Risk  exposure  analysis  complements  and  extends  traditional  cost  and  schedule  risk  analysis. 
Traditionally,  risks  in  cost,  schedule  and  technical  progress  in  engineering  and  manufacturing 
development  have  been  addressed  in  separate  "stovepipes.  Schedule  risk  analysis  did  not  inform 
technical  risk  analysis,  etc.  Our  approach  to  Risk  exposure  analysis  using  risk  leading  indicators 
and  risk  estimating  relationships  provides  an  integrated  approach. 
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Significant  developments  are 

1.  Specifying  risk  leading  indicators  related  to  root  causes  of  adverse  acquisition  outcome 
and  detectable  in  program  documentation 

2.  Producing  a  structure  and  method,  within  the  context  of  standard  acquisition 
documents  and  contract  data  reporting  items,  to  provide  integrated  risk  exposure 
analysis  with  pointers  to  technology/functional  component  areas,  and  to  root  causes 

3.  Specifying  a  process  to  calibrate  the  expected  impacts  in  the  context  of  the  specific 
program  under  consideration. 

Further  research  and  development  plans  are  to  demonstrate,  test  and  evaluate  the  risk  leading 
indicators  and  risk  exposure  calibration  methods  to  assess  practicality,  feasibility,  relevance  and 
significance  via  application  to  acquisition  programs  pre-  and  post-EMD  award. 
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Introduction 


The  project  goal  is  to  develop,  verify,  validate,  assess,  and  transition  practical,  relevant,  and 
significant  methods,  procedures,  and  tools  (MPT)  that  provide  objective  and  quantified  leading 
indicators  of  acquisition  program  cost,  schedule,  and  performance  risks.  The  methods, 
procedures  and  tools  are  intended  to  complement,  not  replace,  current  risk  management 
practices  by  providing  early  warning  of  risk  exposure,  i.e.,  conditions  that  tend  to  increase  the 
likelihood  and/or  severity  of  adverse  acquisition  outcomes. 

The  primary  products  are  (1)  a  set  risk  leading  indicators  (RU),  i.e.,  objective  metrics  calculated 
from  standard  program  documents  and  contract  data  reporting  items  that  provide  early  warning 
of  areas  of  risk  exposure,  and  (2)  methods  to  calibrate  and  baseline  the  relationships  between 
the  risk  leading  indicators  and  program  outcomes.  Secondary  products  are  (3)  recommendations 
for  Request  for  Proposal  (RFP)  language  to  improve  visibility  and  transparency  of  risk  exposure 
during  Engineering  and  Manufacturing  Development  (EMD),  and  (4)  recommendations  for 
collecting  and  sharing  data  to  calibrate/baseline  risk  estimating  relationships  across  programs. 
The  risk  leading  indicators  point  to  root  causes,  and  indicate  the  technical  areas  of  risk  in  the 
program. 

The  risk  leading  indicators  address  (1)  evidence  of  risk  exposure  prior  to  EMD  award,  and  (2) 
evidence  of  risk  exposure  during  EMD  execution.  The  risk  leading  indicators  are  intended  to  be 
used  by  Government  analysts  and  risk  managers,  and  by  EMD  contractors  in  their  risk 
management  program.  The  risk  leading  indicators  are  being  co-developed  working  with  end- 
users,  and  are  being  validated  and  assessed  by  application  to  a  major  defense  acquisition 
program,  in  concert  with  the  program  Risk  Manager.  This  report  describes  progress  and  results 
during  2014,  focused  on  developing  and  verifying  risk  leading  indicators,  and  plans  for  2015, 
focusing  on  pilot  application  and  evaluation  in  cooperation  with  a  ground  combat  major  defense 
acquisition  program  (MDAP). 

The  steps  of  the  technical  approach  are 

1.  To  develop  provisional  RU  by  building  on  prior  NDAI/INCOSE  work  on  system 
development  leading  indicators,  GAO  "best  practices",  and  prior  research  by  IDA,  MITRE 
and  others  on  root  causes  of  adverse  acquisition  outcomes  and  their  causal  mechanisms 

2.  To  derive  RU  from  the  requirements  and  evaluation  criteria  in  RFP  packages  in  order  to 
ensure  that  the  RU  that  reflect  concerns  and  evidence  of  program  progress  from  the 
perspective  of  the  program  management  organization 

3.  To  cross-reference  the  RU  to  program  documents  and  contract  reporting  data  items,  in 
order  to  ensure  that  the  data  are  will  be  available  to  calculate  the  risk  leading  indicators 

4.  To  verify  the  suitability  and  credibility  of  the  RU  by  co-developing  them  with  end-users, 
specifically  the  TARDEC  Risk  Management  Group,  and  by  periodic  technical  exchange 
meetings  with  other  end-users  (e.g.,  the  US  Army  Materiel  Systems  Analysis  Agency, 
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AMSAA),  theoreticians  at  RAND,  MITRE,  IDA,  and  academics,  and  discussions  with  DASD 
personnel 

5.  To  develop  methods  to  calibrate  the  RU  (a)  to  any  specific  program  using  contract 
reporting  data  on  program  cost,  schedule  and  performance  progress,  and  (b)  across 
programs  using  such  reporting  data  as  are  archived  and  shareable  across  programs 

6.  To  validate  and  evaluate  the  RU  by  pilot  application  to  a  major  defense  acquisition 
program,  working  in  coordination  with  the  program  Risk  Manager  and  the  TARDEC  Risk 
Management  team 

7.  To  facilitate  transitioning  the  methods,  procedures  and  tools  to  the  end-user  community 
by  working  with  them  to  develop  and  pilot  the  methods. 

Activities  in  2014  focused  on  steps  1-4,  developing  provisional  RU.  The  primary  focus  for  2015 
will  be  on  steps  5-7,  piloting  the  RU.  In  2014,  we  reached  an  agreement  with  a  MDAP  to  pilot  the 
RU  in  support  of  program  risk  management,  in  a  collaboration  between  SERC  researchers,  the 
TARDEC  risk  management  team,  and  the  program  risk  manager.  However,  the  program  was  in 
source  selection  for  the  EMD  phase  until  late  December  2015.  During  this  time  there  was 
turnover  in  the  PMO.  We  decided  that  we  needed  to  reaffirm  the  collaboration  agreement  and 
had  a  joint  meeting  in  February  2015  to  re-connect.  In  February  2015  we  initiated  discussions 
with  another  ground  combat  program  ("Future  Tank")  that  is  earlier  in  the  acquisition  life-cycle 
to  pilot  pre-EMD  RU. 

The  pilot  applications  to  demonstrate  the  value  of  the  RU  and  facilitate  transition  into  use  are 
the  primary  focus  for  2015.  We  also  plan  to  pursue,  at  a  low  level  of  effort,  RU  specifically 
formulated  for  software-intensive  cyber-physical  systems  with  real-time,  distributed,  embedded 
processing  integrated  with  sensors  and  actuator  electronics. 


Scope  and  Objectives 

The  primary  2014  objectives  were  to  develop  practical,  relevant,  and  significant  leading 
indicators  of  risk  exposure,  i.e.,  program  conditions  that  amplify  the  likelihood  and/or 
consequences  of  unforeseen  future  events  and  interactions,  and  of  normal  variances  in  activity 
time,  cost  and  performance.  Adverse  consequences  include  development  time  and  cost  overrun, 
technical  performance  and  reliability  shortfall,  and  excessive  production,  operation,  and 
sustainment  costs.  Risk  exposure  is  the  result  of  unrecognized  or  unacknowledged  past  events, 
decisions  and  actions,  and  current  uncertainty,  inconsistency,  incompleteness,  and 
interdependency  in  the  system  development. 

Risk  exposure  early  warning  refers  to  detecting  evidence  of  the  root  causes  and  effects  before 
there  are  significant  adverse  consequences.  Its  purpose  is  to  alert  program  management  to  areas 
of  elevated  risk  exposure  in  the  program.  Risk  exposure  early  warning  combines  program  cost, 
schedule  and  performance  data  in  an  integrated  view. 
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Risk  exposure  early  warning  complements  the  risk  identification  practices  and  procedures  in  the 
DoD  Risk  Management  Guide  [1],  The  intent  of  the  Guide  is  "to  help  ensure  program  cost, 
schedule,  and  performance  objectives  are  achieved  at  every  stage  in  the  life  cycle"  and  to  present 
processes  "for  uncovering,  determining  the  scope  of,  and  managing  program  uncertainties." 

The  Risk  Management  Guide  defines  a  risk  as  a  potential  future  event  which,  if  it  occurs,  would 
have  adverse  consequences,  for  which  the  probability  that  it  will  occur  and  the  consequences,  if 
it  occurs,  can  be  assessed.  Risks  with  high  combined  likelihood  and  magnitude  are  priorities  for 
tracking  and  mitigation.  The  practices  and  procedures  in  the  guide  start  with  identifying  risk 
events.  Risk  exposure  early  warning  does  not  itself  identify  causes  or  mitigations.  It  detects 
conditions  of  elevated  exposure  to  risk  and  it  points  to  evidence  to  help  orient  risk  and  issue 
management  investigation. 

The  procedures  and  practices  in  the  Guide  rely  on  Subject  Matter  Experts  (SMEs)  to  identify  and 
quantify  risk  events.  Risk  exposure  early  warning  provides  an  evidence-based  analytic  approach 
that  complements  SME  insight.  The  Air  Force  Cost  Risk  Handbook  [2]  identifies  common  factors 
leading  to  bias  and  dispersion  in  SME  estimates  of  time,  cost,  technical  performance,  and  of 
identification  and  estimation  of  likelihood  and  consequences  of  risk  events.  It  lists  eight 
motivational  factors  (management  pressure,  social  pressure,  group  think,  wishful  thinking,  career 
goals,  misunderstanding,  project  advocacy,  and  competitive  pressures)  and  five  cognitive  bias 
factors  (inconsistency  over  time,  anchoring,  irrelevant  analogies,  underestimation,  and  human 
nature). 

Risk  exposure  early  warning  is  based  on  understanding  the  causal  chains  from  root  causes  to 
adverse  outcomes,  and  how  the  effects  manifest  in  standard  program  and  system  development 
data  and  reports.  Risk  exposure  early  warning  has  two  major  components:  Risk  Leading 
Indicators  (RU),  and  Risk  Estimating  Relationships  (RER). 

RU  are  computed  from  standard  program  documentation  and  contract  data  reporting  items 
(CDRI).  RU  compare  data  across  different  reporting  requirements  and  over  time  to  measure 
incompleteness  and  inconsistency,  instability,  uneven  or  inadequate  progress,  and 
interdependency  of  the  program  and  system  organization.  The  RU  are  evidence  of  both  (1)  root 
cause  problems  with  potential  for  persistent  future  effects,  and  (2)  conditions  that  increase  the 
potential  for,  and/or  sensitivity  to,  unforeseen  future  events  and  execution-versus-plan 
variances.  The  RU  were  developed  through  examination  of  risk  considerations  in  proposal 
evaluation  and  program  execution  criteria  and  reporting,  published  root  cause  analyses,  best 
practices  metrics  for  schedule  risk  analysis,  and  published  research  on  system  development 
leading  indicators. 

Risk  Estimating  Relationships  (RER)  are  statistical  models  that  use  the  RU  to  estimate  future  bias 
and  uncertainty  in  program  activities  time,  cost  and  technical  performance.  The  RER  are 
calibrated  to  data  from  completed  Integrated  Master  Schedule  (IMS)  activities  of  the  current 
program,  and  are  updated  over  time.  Each  completed  activity  in  the  IMS  provides  a  data  point. 
RER  can  be  as  simple  as  thresholds  for  "high"  and  "low"  risk  exposure,  or  as  complex  as  regression 
models  to  forecast  the  likelihood  and  expected  magnitude  of  cost  and  schedule  overrun,  or 
system  performance  shortfall.  System  performance  shortfall  is  evidenced  by  capabilities 
deferred  to  future  product  improvements,  and/or  reduced  capability  goals. 
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The  research  is  organized  into  two  parts  based  on  the  DoD  acquisition  lifecycle:  (1)  pre-EMD 
award,  and  (2)  post-EMD  award.  Prior  to  EMD  award,  the  Government  has  the  authority  and 
responsibility  to  define  the  parameters  of  the  EMD  program.  After  EMD  award,  the 
Government's  role  is  largely  one  of  monitoring  and  evaluation,  leading  up  to  the  Milestone  C 
decision  whether  or  not  to  proceed  to  production.  Up  to  the  point  of  EMD  contract  award,  the 
Government  frames  the  acquisition  program  in  terms  of  the  system  performance  and  cost,  and 
the  EMD  time  and  cost.  After  EMD  contract  award,  the  Government  and  contractor  are  bound 
by  the  terms  of  the  award,  subject  to  negotiated  changes.  From  the  system  acquisition 
perspective,  there  are  risks  embedded  in  the  program  prior  to  EMD  award,  and  other  risks  in  the 
EMD  execution. 

The  scope  of  this  project  excludes  risks  of  Congressional  decisions  (funding  of  the  program  of 
interest,  and  competing  programs/plans;  cost  impacts  of  location  of  R&D  and  production,  etc.). 
The  scope  excludes  risks  of  long  time  to  fielding  and  slow  rate  of  production.  These  are 
Government  decisions  driven  by  considerations  outside  the  scope  of  program  time/cost/risk 
capability. 


Approach 

We  began  by  reviewing  prior  research  pertinent  to  developing  RLI.  We  reviewed  prior  analyses 
of  root  causes  of  adverse  program  outcome.  We  identified  Program  Management  Office  (PMO) 
risk  issues  and  perspectives  expressed  and  implied  in  the  evaluation  criteria  and  reporting 
requirements  in  Request  for  Proposal  (RFP)  packages  over  several  ground  vehicle  programs.  We 
reviewed  reports  on  system  development  leading  indicators.  We  reviewed  metrics  from 
recommended  procedures  and  "best  practices"  for  program,  cost  and  schedule  management. 

We  then  compared  the  potential  indicators  and  metrics  to  sources  of  data  and  evidence  in 
program  management  and  systems  engineering  baseline  and  update  reports,  and  contractor 
data  reporting  items.  We  then  defined  an  initial  set  of  evidence-based  risk  leading  indicators  that 

(1)  can  be  computed  from  computable  from  data  in  typical  program  management  and 
systems  engineering  reports  and  updates 

(2)  address  significant  root  causes  of  adverse  acquisition  outcome 

(3)  indicate  the  technical  areas  of  the  program  contributing  significantly  to  the  risk  exposure 
for  risk  management  review  and  mitigation. 

We  examined  issues  in  using  data  from  previous  programs  to  calibrate  cost  estimating 
relationships  as  a  proxy  for  risk  estimating  relationships.  We  examined  the  availability  and 
accessibility  of  data  from  past  programs  to  use  to  calibrate  risk  estimating  relationships.  We 
concluded  that  access  to  relevant  calibration  data  from  past  programs  would,  in  many  cases,  be 
difficult  if  not  impossible.  Therefore  we  developed  the  outline  of  a  method  to  use  current 
program  data  to  self-calibrate  the  risk  estimating  relationships  to  predict  the  probability  and 
magnitude  of  time  and  cost  overrun. 
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Prior  Research  Foundations 


Root  Cause  Analyses 

The  Government  Accountability  Office  (GAO)  [3]  reported  that  in  fiscal  year  2012,  of  the  85  major 
defense  acquisition  programs  under  review,  39  percent  had  unit  cost  growth  of  25  percent  or 
more,  the  average  delay  in  initial  operating  capability  was  27  months,  the  average  change  in 
development  cost  from  the  initial  estimate  was  49  percent,  and  the  average  change  in  total 
acquisition  cost  from  the  initial  estimate  was  38  percent.  The  report  finds  that  programs 
experiencing  cost  and  schedule  growth  " share  a  common  dynamic:  moving  forward  with 
programs  before  the  knowledge  needed  to  make  decisions  is  sufficient."  The  report  identifies 
seven  common  root  causes:  (1)  concurrent  testing  and  production,  (2)  optimistic  assumptions, 
(3)  delayed  testing,  (4)  insufficient  tradeoffs  among  cost,  schedule,  and  technical  performance 
requirements  during  early  planning,  (5)  unrealistic  cost  and  schedule  estimates,  (6)  Insufficient 
testing  during  development,  (7)  insufficient  attention  to  reliability. 

The  GAO  [4]  identified  12  root  causes:  unstable  program  requirements,  funding  and  quantities; 
complex  systems;  diminishing  industrial  base;  new  processes;  immature  or  cutting  edge 
technologies;  first  time  integration;  unrealistic  assumptions  and  projections;  overoptimistic 
program  baselines;  inexperienced  staff;  lack  of  relevant  historical  data;  unreliable  Earned  Value 
Management  (EVM)  data;  and  inadequate  contingency  schedule  slack  and  management  reserve 
funds. 

The  Office  of  Performance  Assessments  and  Root  Cause  Analyses  (PARCA)  [5]  found  five  root 
causes  of  adverse  acquisition  outcomes  attributable  to  planning  and  execution:  (1)  unrealistic 
cost  or  schedule  estimates,  (2)  immature  technology  with  excessive  manufacturing  and 
integration  risk,  (3)  unrealistic  performance  expectations,  (4)  unanticipated  design,  engineering, 
manufacturing  or  technology  issues,  and  (5)  poor  management  performance.  Poor  management 
performance  included  ambiguities  combining  requirements  and  requirements  documents, 
interface  management,  tradeoffs  within  holistic  performance  attributes  (size,  weight,  power, 
cooling,  etc.),  and  risk  assessment. 

In  2008,  the  NDIA  Systems  Engineering  Division  in  conjunction  with  DASD-ATL-SE  produced  a 
report  on  the  systemic  root  causes  of  program  failures  [6]  concluding  that  "the  most  significant 
causes  were  directly  related  to  poor  or  inadequate  activities  early  in  acquisition  strategizing  and 
planning  efforts  and  in  conducting  management  gate  reviews  during  the  early  stages  of 
execution.  Lastly,  the  analysis  also  concluded  that  there  was  a  significant  root  cause  related  to 
staff  size,  training  and  experience." 

The  Defense-Industrial  Initiatives  Group  at  the  Center  for  Strategic  &  International  Studies  [7] 
found  that  of  85  major  programs,  overoptimistic  estimates  were  the  primary  driver  for  cost 
growth  and  changes  in  quantity  were  the  second  leading  cause.  A  RAND  study  [8]  found  that  in 
their  analysis  of  35  programs,  changes  in  quantity  accounted  for  more  than  half  of  cost  growth, 
and  that  "decisions  to  change  the  schedule,  additional  requirements,  and  cost-estimating  errors 
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account  for  almost  all  of  the  remaining  procurement  cost  growth."  The  Institute  for  Defense 
Analysis  [9]  found  that  " Virtually  every  program  we  surveyed  experienced  cost  problems  that 
could  have  been  avoided  or  ameliorated  through  betterfront-end  analysis  of  overall  design  issues 
and  risks.  Serious  attention  to  system-level  risk  seems  to  have  been  lacking  on  the  part  of  senior 
decision  makers." 

In  summary,  root  causes  include:  unstable  requirements;  incomplete  or  unstable  planning; 
overoptimistic  or  inaccurate  estimating;  underappreciated  or  misunderstood  technical  and 
engineering  challenges;  deferred  or  insufficient  verification  and  testing  during  development; 
unforeseen  interactions  and  interdependencies  in  the  system  requirements,  architecture  and 
design;  deferred  tradeoff  decisions;  unforeseen  interactions  and  interdependencies  in  the 
execution  task  schedule  or  organization;  lack  of  "margin  for  error"  in  budget,  schedule, 
performance  and  design;  data  quality  and  availability  of  relevant  historical  data. 


Proposal  Evaluation  and  Contract  Management 

Fair,  equal  and  open  competition  guidelines  require  that  RFP  packages  clearly  state  (1)  what 
information  to  provide  in  the  proposal  (section  L),  (2)  how  the  proposals  will  be  evaluated 
(section  M),  (3)  the  Scope  of  Work  (section  C),  and  the  Contract  Deliverable  Requirements  List 
(CDRL;  typically  Appendix  A).  The  RFP  package  typically  includes  attachments  that  specify 
documentation  frameworks  and  criteria.  The  CDRL  section  specifies  requirements  for  reports 
and  updates  on  periodic  or  event-based  timelines,  and  the  required  format  and  content. 
Baselines  are  required  either  with  the  proposal,  or  at  a  specific  subsequent  technical  review 
event.  The  RFP  package  typically  includes  guidance  for  duration  of  TD  and  EMD  phases,  funds 
available  for  each  phase,  number  of  EMD  prototypes,  number  of  Low  Rate  Initial  Production 
units,  etc.  The  RFP  package  also  includes  targets  for  performance  (requirements)  and 
constraints. 

Section  M  specifies  how  the  Government  will  evaluate  the  risk-vs-reward  to  reject  unsuitable 
bids,  and  to  select  from  among  suitable  bids.  Risks  are  the  risks  of  failing  to  deliver  prototypes 
that  will  pass  Operational  Testing  (per  specified  failure  definitions  and  scoring  criteria)  for 
performance  and  reliability  on  schedule  and  within  development  budget,  of  failing  to  be  ready 
for  Low  Rate  Initial  Production  (LRIP)  at  the  end  of  the  EMD,  of  failing  to  meet  LRIP  unit  cost 
targets,  and/or  failing  to  be  able  to  meet  fuel  economy,  reliability  and  logistics  support  goals. 
Rewards  are  the  proposed  time,  costs,  and  system  performance.  Risk-reward  tradeoffs  are 
considered  both  in  proposal  evaluation  and  in  program  management  decisions  to  trade  lower 
performance  for  lower  cost-schedule  risk,  for  increased  design  tradespace,  and/or  for  increased 
reliability. 

The  level  of  risk  in  the  proposed  program  is  judged  based  on  the  claimed  level  of  design, 
manufacturing,  production  cost,  and  RAM  maturity,  considering  the  thoroughness  and  credibility 
of  the  supporting  data.  Maturity  levels  are  clearly  defined  with  specific  completion  criteria  based 
on  systems  engineering,  design,  integration,  analysis,  manufacturing,  and  testing  artifacts.  The 
specific  criteria  for  maturity  levels  combine  essential  technical  performance  measures,  systems 
engineering  and  design  technical  review  knowledge  points,  and  verification  results.  The  highest 
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levels  of  maturity  correspond  to  completion  of  the  contract  requirements  with  tests  completed 
on  manufactured  systems,  with  substantiating  cost  data  and  correct  action  plans  for  any 
deficiencies. 

Maturity  level  advancements  are  progress  accomplishments  on  the  path  to  successful  program 
completion.  Using  maturity  levels  to  track  technical  progress  is  a  well-defined  and  is  consistent 
with  the  PMO  framework  for  program  risk  assessment.  Maturity  level  advancement  vice  the 
program  plan  is  a  potential  indicator  of  risk.  Maturity  level  advancement,  over  the  entire 
program  and  by  WBS  element,  can  be  integrated  into  the  scheduling  and  reporting  framework 
simply  by  including  maturity  advancement  in  the  IMP,  so  that  IMP  events,  accomplishments  and 
criteria  include  maturity  advancement  of  the  design,  manufacturing,  and  RAM.  Making  maturity 
advancement  events  part  of  the  IMP  forces  cost  and  schedule  reporting  to  be  reported  relative 
to  demonstrated  maturity.  Maturity  levels  are  organized  by  stages  of  system  acquisition: 
development,  production,  and  operation  &  sustainment.  Maturity  level  assessments  take  a  life 
cycle  cost  and  capability  view.  Technology,  integration,  and  manufacturing  readiness  views  and 
consistent  perspectives,  focused  on  TD  and  EMD  acquisition  stages. 

The  RFP  package  includes  additional  requirements  for  risk  assessment  in  proposal  evaluation  and 
contract  execution.  Executing  a  risk  management  plan,  per  the  DoD  Risk  Management  Guide,  is 
required  to  identify  and  address  potential  future  events  with  significant  likelihood  of  occurrence 
and  significant  consequences  if  they  do.  Technology  Readiness  Assessment  (TRA)  [10]  is  required 
for  those  technologies  designated  as  Critical  Technology  Elements  (CTE)  and  Other  Technologies 
of  Interest  (OTI).  TRA  involves  detailed,  engineering-level  analysis  by  Subject  Matter  Experts.  It 
is  focused  on  the  development  of  selected  technologies  and  subsystems,  not  the  entire 
development  program.  Integration  Readiness  Assessment  (IRL)  and  Manufacturing  Readiness 
Assessment  (MRL)  are  not  always  required. 

Technical  review  checklists  contain  over  800  specific  questions  in  13  categories  to  gauge  progress 
at  each  of  the  major  program  reviews.  Progress  for  each  question  is  scored  red,  amber,  green, 
unknown,  or  not  applicable.  A  team  at  RAND  has  proposed  using  the  technical  review  checklists 
as  the  basis  for  risk  assessment  [11],  The  categories  are  review  entry  readiness,  planning, 
schedule,  management,  staffing,  process,  product  support,  requirements  management,  system 
design,  system  verification,  program  risk  assessment,  certification  and  legal,  and  review 
completion. 

Schedule  risk  assessment,  e.g.,  using  the  Defense  Contract  Management  Agency  (DCMA)  14- 
point  schedule  assessment,  GAO  schedule  assessment  or  similar  approach,  is  typically  required. 
Methods  for  schedule  risk  assessment,  and  the  understanding  of  critical  paths,  near-critical  paths 
and  high  schedule  risk  activities  in  non-deterministic  programs,  are  evolving. 

The  RFP  package  specifies  the  program  management  and  system  engineering  baselines  and 
update  reports,  including  content,  format,  and  frequency  or  event  timing.  These  provide  the 
basis  to  evaluate  risk  leading  indicators  and  to  calibrate  risk  estimating  relationships. 
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System  Development  Leading  Indicators 


The  NDIA  System  Engineering  Division  [12]  found  that  “Technical  decision  makers  do  not  have 
the  right  information  &  insight  at  the  right  time  to  support  informed  &  proactive  decision  making 
or  may  not  act  on  all  the  technical  information  available  to  ensure  effective  &  efficient  program 
planning,  management  &  execution".  The  NDIA  formed  a  working  group  to  develop  a  set  of 
system  development  leading  indicators  to  provide  insight  into  technical  performance  at  major 
decision  points  for  managing  programs  quantitatively  across  their  life  cycle,  with  emphasis  on  TD 
and  EMD  phases,  and  objective  measures  of  commonly  and  readily  available  data. 

The  NDIA  project  used  surveys  to  identify  high-value  areas  for  system  development  leading 
indicators,  building  on  prior  work  on  systems  engineering  leading  indicators  [13].  The  system 
development  leading  indicator  categories  [14,  15]  were: 

1.  Requirements  Stability 

2.  Proportion  of  Stakeholder  Needs  Met  and  Verified 

3.  Interface  Completion  Trends 

4.  Staffing  Skills  and  Trend 

5.  Risk  Burndown 

6.  Technical  Performance  Measure  (TPM)  Trends 

7.  Technology  Readiness  Level 

8.  Manufacturing  Readiness  Level 

9.  Architecture 

10.  Affordability 

11.  Requirements  Verification 

12.  Defects  and  Errors 

Specific  leading  indicators  were  defined  in  some  of  the  categories.  The  indicators  were  not 
directly  tied  to  specific  contract  data  requirements.  Typical  contract  data  requirements  relate  to 
categories  #1,  2,  5,  7,  8,  9,  10  &  11.  Proposal  evaluation  and  contract  management  criteria 
suggest  additional  indicators,  supported  by  specific  technical  status  and  progress  reporting,  and 
directly  related  to  program  risk  considerations  in  program  decisions. 

The  NDIA  survey  referenced  Kohl  and  Carson  [16]  reporting  on  a  Practical  Software  &  System 
Measurement  workshop  on  architecture  measurement  concepts.  Although  the  workshop 
focused  on  software  and  system  architecture,  the  concept  of  architecture  also  applies  to  the 
program  architecture  as  expressed  in  the  IMS,  to  the  requirements  architecture,  and  other 
knowledge  structures  in  program  management  and  systems  engineering.  The  workshop 
consensus  identified  six  dimensions  of  architecture  development  for  measurement:  size, 
interconnectedness,  completeness,  compliance,  consistency,  and  cost.  "Stability"  was  not 
included.  Sources  of  input  data  and  methods  to  calculate  measures  from  the  data  were  not 
specified. 

The  Defense  Contract  Management  Agency  (DCMA)  approach  to  technical  performance  risk 
assessment  evaluates  variances  of  technical  progress  versus  cost  and  schedule  to  indicate  the 
level  of  risk  and  detect  new  risks  before  their  effects  on  cost/schedule  are  irrevocable  [17,  18]. 
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Technical  progress  is  measured  by  the  Technical  Performance  Measures  (TPM)  for  the  Technical 
Performance  Parameters  (TPP).  Cost  and  schedule  are  measured  with  the  Earned  Value 
Management  (EVM)  system.  The  TPP  and  TPM  are  formulated  for  the  particular  program,  and 
are  derived  from  the  major  system  performance  parameters.  In  this  model,  TPM  progress  against 
the  plan  is  the  basis  for  leading  indicators  of  risk. 


Best  Practices  for  Schedule  Risk  Assessment 

The  GAO  schedule  risk  assessment  guide  [19]  addresses  measurement  of  schedule  health  with 
quantitative  approaches  to  analyze  the  completeness,  consistency,  interdependency,  safety 
margins  in  a  program  schedule.  The  metrics  include:  the  number  and  proportion  of  "long" 
activities;  ratio  of  activities  to  dependency  links  in  total  and  by  each  of  the  four  types  of  logical 
dependency;  ratio  of  detailed  activities  to  milestones;  number  and  proportion  of  activities  no 
mapped  to  a  milestone  or  Integrated  Master  Plan  (IMP)  event;  number  and  proportion  of 
activities  with  many  predecessor  links,  with  many  successor  links,  with  no  successor,  and  with  no 
predecessor;  critical  path  float  to  each  milestone;  and  number  of  activities  with  negative  or  low 
float  relative  to  their  planned  duration.  The  guide  also  includes  of  schedule  execution,  e.g., 
number  of  activities  that  were  started  or  finished  before  their  logical  dependency  condition, 
number  of  activities  that  started  or  finished  late,  mean  and  standard  deviation  of  the  difference 
between  actual  and  planned  duration,  start  date,  and  end  date. 

The  GAO  Guide  recommends  using  probabilistic  schedule  risk  analysis  to  complement 
deterministic  critical  path  analysis.  Probabilistic  risk  analysis  requires  data  on  the  distribution  of 
activity  duration,  e.g.  the  minimum,  most  likely,  and  maximum,  and  data  on  the  correlations 
between  activity  durations.  Simulation  is  used  to  compute  the  probability  distribution  of  total 
time.  The  probability  the  program  will  be  late  or  milestone  missed,  and  the  expected  amount 
late  given  it  is  late,  are  computed  from  the  distribution.  However,  no  published  data  has  been 
found  showing  that  people  can  reliably  estimate  the  distribution  of  activity  durations  or  the 
correlations  between  activities.  A  detailed  IMS  can  have  upwards  of  5,000  activities,  and 
providing  probability  distribution  and  correlation  estimates  can  be  onerous.  Probabilistic  risk 
analysis  does  not  directly  identify  which  activities  are  putting  the  schedule  most  at  risk.  Further 
development  and  verification  of  practical  methods  for  schedule  risk  analysis  are  needed. 


Statistical  Estimating  Relationships 

The  concept  of  RERs  was  inspired  by  the  statistical  approach  to  Cost  Estimating  Relationships 
(CERs).  CERs  estimate  cost  by  calibrating  a  model  of  historical  cost  data  as  a  function  of  a  set  of 
explanatory  factors  [2,  20,  21].  The  proportion  of  cost  variance  explained  by  a  CER  is  a  measure 
of  the  accuracy  of  the  model.  Open  questions  in  developing  CERs  include:  choosing  the 
population  of  "similar"  cases  to  pool  together;  choosing  the  explanatory  factors;  choosing  the 
general  analytic  model  type.  These  choices  are  interrelated.  Larger  pools  of  more  diverse 
programs  may  provide  more  or  less  consistent  evidence  of  different  interactions  and 
dependencies.  When  the  ratio  of  data  points  to  explanatory  variables  and  model  parameters  is 
low,  there  is  a  risk  of  spurious  correlation.  Some  explanatory  variables  may  be  highly  correlated. 
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or  correlated  in  some  programs  but  not  others.  Data  from  the  program  of  interest  may  be  sparse, 
but  is  highly  relevant. 

The  significant  issues  in  formulating  CERs  are  (1)  choice  of  the  systems  to  pool  as  a  population  of 
similar  cases,  balancing  population  size  and  diversity,  (2)  choice  of  the  explanatory  factors,  i.e., 
independent  variables,  (3)  choice  of  the  underlying  regression  technique.  Of  these  three  issues, 
the  first  two  are  the  most  important.  In  practice,  limitations  in  data  quality  and  quantity  limit 
CER  accuracy,  more  than  differences  between  regression  techniques. 

Pooling  over  a  diverse  population  attenuates  and  obscures  the  effects  of  individual  factors  and 
interaction  effects.  A  small  population  risks  biases  from  small  sample  size.  Pooling  multiple 
programs  is  needed  to  reduce  statistical  uncertainty,  but  overbroad  pooling  increases 
uncertainty  [22], 

CERs  estimate  cost.  The  regression  for  CER  calibration  produces  an  estimate  of  the  accuracy  of 
the  model,  i.e.,  the  proportion  of  variance  in  the  sample  data  explained  by  the  model  and  the 
expected  error  in  the  model  predictions.  But  CERs  do  not  attempt  to  estimate  the  dispersion  in 
cost  as  a  function  of  the  input  variables.  They  estimate  cost,  but  not  cost  variability. 

RERs  are  an  analog  to  CERs,  but  with  the  addition  of  estimating  variability  as  well  as  expectation. 
Unexpected  adverse  outcomes  result  from  bias  in  the  program  estimates  of  time,  cost  and 
performance  and  uncertainty  (also  called  dispersion)  in  the  difference  between  estimates  and 
actuals.  RERs  explain  the  accuracy  of  time/cost/performance  planned  versus  actual  outcomes. 
Accuracy  has  two  components:  (a)  bias  or  offset,  and  (b)  random  dispersion  or  uncertainty. 

Risk  sources  may  differ  from  program  to  program,  from  acquisition  stage  to  stage,  and  between 
WBS  elements.  Sources  of  risk  exposure  can  be  highly  varied  between  different  individual 
programs  of  the  same  type  due  to  a  wide  variety  of  factors  including  marketing  strategy, 
experience  of  the  engineering  and  engineering  management  team,  aggressiveness  of  the  cost, 
schedule  and  performance  goals,  technology,  engineering,  and  integration  challenges,  etc. 
Different  program  can  have  different  reporting  requirements  and  interpretations.  Above  and 
beyond  the  difficulty  of  sharing  data  across  programs,  the  diversity  of  programs  is  a  major 
obstacle  to  calibrating  RERs  to  past  program  data. 

An  alternative  is  to  calibrate  the  RERs  to  data  from  the  individual  program.  This  ensures 
relevance.  The  limitation  is  that  at  the  start  of  the  program,  there  is  no  data.  As  the  program 
proceeds  and  data  are  collected,  the  RERs  can  be  calibrated.  To  obtain  statistically  valid  samples, 
calibration  data  points  should  be  at  the  lowest  level  of  the  IMS  with  time  and  cost  reporting  - 
each  completed  IMS  activity  is  data  point,  conditioned  on  being  part  of  the  same  program. 
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Findings  and  Results 


Causal  Mechanisms 

This  section  presents  a  synthesis  of  the  published  analyses  on  root  causes,  acquisition,  program 
and  contract  management,  and  system  development,  augmented  by  first-hand  experience  on 
acquisition  program  execution  and  Independent  Review  Teams.  The  observations  in  this  section 
are  the  rationale  for  the  choices  of  risk  leading  indicators  and  the  integrated  risk  early  warning 
approach. 

Optimistic  Estimates  Have  Real  Consequences 

"Success  oriented"  program  plans  create  skewed  distributions  with  long  tails:  appealing  planned 
results,  but  with  greater  adverse  consequences  when  problems  occur.  In  an  effort  to  "sell"  a 
program  and/or  to  win  a  bid,  senior  management  can  be  tempted  to  make  optimistic  claims  for 
time,  cost,  technical  performance,  reliability,  and  potential  risks.  Since  adverse  acquisition 
outcomes  are  deficiencies  relative  to  how  the  program  was  sold,  optimistic  claims  increase  risk 
by  reducing  the  margin  for  error  and  uncertainty. 

Beyond  this  "statistical"  effect,  optimistic  estimates  have  real  effects  on  the  program  plans  that 
lead  to  elevated  risk  exposure.  An  optimistic  timeline  leads  to  aggressive  schedule  structures. 
Characteristics  of  an  aggressive  schedule  structure  include:  concurrent  (parallel  but  independent) 
paths  that  come  together  towards  the  end;  limited  incremental  integration  analysis,  verification 
and  testing;  low  schedule  slack  margin  for  error;  and  time  estimates  that  are  more  optimistic  for 
activities  later  in  the  schedule.  Programs  concurrent  paths  and  limited  intermediate  integration 
and  verification  ("it  all  comes  together  at  the  end")  do  not  produce  the  information  to  detect  and 
correct  problems  until  it  is  too  late.  Aggressive  cost  goals  lead  to  reduced  and  deferred 
developmental  analysis  and  testing,  and  to  eliminating  parallel  execution  of  alternative  backup 
approaches.  Aggressive  performance  goals  lead  to  adopting  less  mature  advanced  technologies 
that  are  more  likely  to  have  unforeseen  integration  issues.  Aggressive  schedules  lead  mid-level 
managers  and  engineers  to  take  shortcuts.  Over  optimistic  goals  have  real  effects  that  elevate 
exposure  to  risk. 

Allowing  Margin  For  Error  Reduces  Risk  Exposure 

Limited  margin  for  error  creates  exposure  to  risk.  When  there  is  little  or  no  safety  margin,  small 
unforeseen  events  and  "natural"  variances  due  to  uncertainty  can  have  amplified  effects.  When 
there  is  more  safety  margin,  larger  impacts  are  needed  to  produce  adverse  consequences. 
Schedule  margin  (also  called  slack  or  float)  covers  schedule  slip  are  rework  time.  Cost  margin 
(management  reserve)  covers  unforeseen  costs  and  gives  management  flexibility.  Performance 
margin  provides  tolerance  for  unforeseen  interactions  that  could  degrade  overall  capability. 
Design  margins  for  holistic  system  properties  such  as  size,  power,  weight,  cost,  reliability,  etc. 
provide  tolerance  for  unforeseen  growth  in  burdens.  The  GAO  recognized  the  need  for 
management  reserve  and  schedule  slack.  Tradeoffs  between  performance  levels  and  holistic 
burdens  are  recognized  in  program  management  and  systems  engineering.  Contract  award 
practices  to  recognize  margins  and  uncertainty  in  cost,  schedule  and  performance  are  evolving. 
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Past  Performance  Predicts  Future  Performance 

The  root  causes  of  time  and  cost  overruns  and  technical  accomplishment  shortfalls  in  IMS 
activities  do  not  go  away  by  themselves.  If  the  planning  was  overoptimistic,  if  technical 
challenges  were  not  well-understood,  if  the  executing  organization  has  internal  problems,  etc., 
and  if  there  has  been  no  underlying  change,  then  the  root  causes  and  dynamics  will  still  be  at 
work,  and  similar  patterns  of  bias  and  dispersion  in  the  outcomes  of  completed  activities  relative 
to  the  plan  are  likely  to  show  up  in  the  future.  In  this  situation  lagging  indicators  become  leading 
indicators. 

Potential  Risks  Are  Everywhere  And  Entangled 

Every  requirement  that  has  not  been  verified,  every  schedule  activity  that  has  not  been 
completed,  every  architecture  element  that  has  not  been  designed,  integrated,  and  tested 
expose  the  program  to  risk.  Decisions  affecting  uncertainty  and  accomplishment  in  one 
acquisition  phase  have  impacts  on  other  phases.  Decisions  that  affect  uncertainty  and 
accomplishment  in  cost,  schedule  and  technical  performance  are  interrelated.  Decisions  and 
tradeoffs  in  one  WBS  or  IMP  element  can  have  impacts  on  others.  Lacking  a  model  and  data  on 
these  interactions  creates  ignorance  that  exposes  the  program  to  risk. 

Evidence  Reveals  Risk  Exposure 

Program  reports  -  baselines  and  updates  of  system  development  data,  linked  to  program 
execution  data  -  are  useful  to  detect  and  diagnose  risk  exposure  when  they  are  timely,  with 
sufficiently  complete,  consistent,  and  accurate  content.  Reporting  and  analysis  of  evidence  has 
costs  that  must  be  considered  relative  to  benefits,  just  as  incremental  integration  and  verification 
has  costs  and  benefits.  Risk  early  warning  can  leverage  standard  reports  and  work  within  the 
framework  of  standard  contract  data  requirements.  The  standard  reports  and  reporting 
schedules  have  been  developed  to  inform  particular  program  stage  gates  and  PMO  decisions,  but 
were  not  specifically  designed  for  integrated  risk  early  warning.  Integrate  risk  early  warning  could 
benefit  from  (1)  update  scheduling  for  proactive  choices  vice  reactive  assessments;  and  (2) 
standardized  content,  format,  and  terms  across  different  artifacts  to  help  ensure  consistency  and 
completeness.  Standardized  language  for  RFP  packages  is  the  mechanism  for  implementation. 
Contractors  will  benefit  from  standardized  contract  language  by  knowing  what  data  content, 
information,  and  terms  to  provide,  and  how  the  data  will  be  assessed.  Increasing  transparency 
to  the  offerors  improves  the  quality  of  responses,  and  the  ability  to  compare  competitor 
responses.  Objective  data  reporting  and  clear  evaluation  criteria  are  essential  for  open  and  equal 
competition  among  vendors 

The  Program  Manager's  Office  (PMO)  Establishes  the  Risk  Tradeoff  Terms  and  Conditions 

The  PMO  is  the  definitive  source  for  understanding  priorities  and  tradeoffs  among  EMD  cost, 
schedule,  technical  performance  &  reliability,  production  and  O&S  cost  and  risks  of  not  meeting 
claims.  The  PMO  determines  the  evaluation  basis  and  criteria  to  compare  the  risk  and  reward  of 
alternative  proposals.  The  PMO  specifies  the  priority  levels  for  different  performance 
parameters  and  constraints.  The  PMO  decides  how  to  trade  off  development  time  and  cost 
versus  initial  performance  and  reliability  versus  production  cost  versus  reliability  growth  and 
performance  upgrade  though  continuous  modernization  versus  operation  and  sustainment  cost 
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and  logistics  footprint.  Risk  exposure  assessment  must  be  consistent  with  the  PMO  value 
proposition  to  be  relevant.  The  same  risk  priorities  and  considerations  apply  after  contract  award 
as  during  proposal  evaluation,  albeit  with  additional  data.  Risk  exposure  early  warning  must  be 
informed  by  and  consistent  with  the  proposal  evaluation  criteria  and  the  contract 
reporting/management  criteria  in  the  RFP  package  produced  by  the  PMO. 

Buried  Tradeoffs  &  Constraints  Indicate  and  Cause  Risk  Exposure 

Sometimes  programs  have  constraints  and/or  tradeoff  relationships  between  one  factor  and 
another  that  have  not  been  included  in  the  product  specification,  e.g.,  size,  power,  weight, 
production  cost,  operational  reliability  and  maintainability,  continuous  modernization  capacity, 
etc.  Sometimes  increasing  performance  on  one  parameter  can  compensate  for,  or  allow  margin 
for,  another.  Decreasing  performance  goals  in  one  area  can  increase  the  design  constraint 
tradespace  over  the  entire  system.  Increasing  clarity  of  the  constraints  &  tradeoffs  reduces  risk 
of  offerors  going  "off  in  the  wrong  direction." 

Instability  Indicates  and  Causes  Risk  Exposure 

Unstable  requirements,  plans,  and  architectures  create  re-work  and  wasted  effort,  and  uncertain 
outcome.  Instability  can  be  evidence  of  inadequate  planning  and  understanding  the  technical 
content,  implying  continued  future  instability.  Changes  create  re-work,  and  indicate  past 
planning  deficiencies  that  will,  if  not  corrected,  lead  to  future  incompatibilities  and  re-work. 

Interdependency  and  Incompletely  Resolved  Elements  Indicate  and  Cause  Risk  Exposure 

Requirements,  system  architectures,  and  program  schedules  are  all  networks  of  interconnected 
nodes.  Larger  and  more  highly  interdependent  networks  have  more  opportunities  for 
unforeseen  interactions,  frictions,  and  ripple  effects.  There  are  more  opportunities  for  adverse 
interactions  if  a  new  node  or  link  is  added.  When  nodes  that  are  not  fully  resolved  (e.g.,  a 
planning  package  in  the  schedule  versus  defined  task,  a  requirement  that  has  not  been 
decomposed  and  linked,  or  an  architecture  element  does  not  have  completed  boundary 
diagrams)  are  eventually  resolved,  there  are  more  opportunities  for  adverse  interactions  in  more 
complex  networks.  Unresolved  elements  are  evidence  of  incomplete  planning  and/or 
understanding  the  system  and/or  program.  Unresolved  elements  add  uncertainty  into  estimates, 
and  may  have  hidden  dependencies.  Unresolved  elements  can  hide  development  obstacles  and 
difficulties,  and  thus  leading  to  overoptimistic  estimates.  The  GAO  recommends  that  no  detailed 
activity  be  longer  than  44  days,  recognizing  that  long  activities  are  evidence  of  incomplete 
resolution  and  that  longer  activities  tend  to  have  greater  uncertainty.  The  GAO  also  recommends 
analyzing  IMS  network  characteristics  to  assess  schedule  risk.  Similar,  though  not  identical, 
methods  can  apply  to  the  requirements  network,  the  system  architecture,  and  even  to  the 
interconnected  network  of  the  requirements,  IMP/IMS,  and  system  architecture. 

Compliance  Verification  and  Developmental  Testing  Buy  Down  Risk 

Less  verification,  incremental  integration  and  testing  during  development  creates  more 
opportunity  for  unhappy  surprises  at  the  end  of  the  program  when  time  and  funds  for  corrective 
actions  are  constrained.  Earlier  verification  provides  more  time  margin  for  corrective  action. 
Verification  and  developmental  testing  require  commitment  of  time  and  funding.  Different 
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means  of  verification  include  design  inspection  and  engineering  judgment,  modeling  and 
simulation,  isolated  bench  testing,  partially  integrated  testing,  and  fully  integrated  testing. 

Technical  Progress  Needs  Visibility 

Reporting  to  reveal  real  technical  progress  is  needed  to  assess  risk.  Without  evidence,  early 
warning  is  unfounded.  Objective  measures,  e.g.,  development,  production  and  RAM  maturity  by 
WBS  element,  provide  diagnostics.  Slower  than  expected  technical  progress  indicates  initial 
planning  bias.  Uneven  technical  progress  indicates  planning  inaccuracy  or  underappreciated 
difficulty.  Incomplete,  inconsistent,  missing  or  "to  be  determined"  fields  in  program 
management  and  system  engineering  baselines  and  reports  are  is  evidence  of  uncertainty,  which 
could  lead  to  new  tasks,  longer  times,  more  coordination,  etc. 

Different  Programs  are  Different 

Different  programs  have  different  engineering  and  manufacturing  challenges,  different 
contractors,  different  engineering  management  practices,  and  different  levels  of  skill  and 
experience.  Different  programs  will  have  different  sources  of  risk.  These  differences  make  it 
difficult  to  extrapolate  from  one  program  to  another.  Risk  Leading  Indicators  that  were  relevant 
to  one  program  may  not  be  relevant  to  another.  Quantitative  relationships  between  leading 
indicators  and  outcomes  on  one  program  may  not  be  accurate  for  another  program.  Aggregating 
across  disparate  programs  would  dilute  and  obscure  the  significant  relationships,  as  seen  in 
empirical  studies  of  Cost  Estimating  Relationships  (CER)  across  different  domains  [22], 


Proposal  and  Contract  Data  For  RLI 

Risk  early  warning  employs  baselines  and  updates  for  seven  standard  data  reporting  elements. 
Three  are  program  management  products:  the  Work  Breakdown  Structure  (WBS),  the  Integrated 
Master  Plan  (IMP),  and  the  Integrated  Master  Schedule  (IMS).  Five  are  systems  engineering 
artifacts:  the  System  Segment  Specification  (SSS),  the  Specification  Tree  (ST),  the  System 
Architecture  (SA),  Technical  Review  Checklists  (TRC),  and  the  Manufacturing  Cost  Estimate  (MCE). 
The  RFP  package  specifies  the  content,  format,  preparation  instructions,  initial  delivery  and 
update  schedules. 

The  Government  provides  the  initial  WBS  in  the  RFP  package  per  MIL-STD-881C  [23],  The 
contract  WBS  has  detail  added  by  the  contractor,  and  is  updated  during  the  program.  The  WBS 
is  the  primary  framework  for  program  organization  and  reporting. 

In  risk  exposure  early  warning,  the  WBS  and  the  IMP  are  the  key  frameworks  to  identify  areas  of 
elevated  risk.  The  initial  IMP  is  prepared  by  the  Government  as  part  of  the  RFP  package.  The 
IMP  is  a  3-level  indented  list  of  events,  accomplishments,  and  criteria.  The  IMP  is  the  basis  for 
IMS  and  Earned  Value  Management  (EVM)  reporting  [24,  25],  EVM  time,  cost  and  progress 
reporting  is  at  the  "Work  Package"  level  of  the  IMS. 

The  high-level  IMS  is  provided  by  the  Government  in  the  RFP  package.  It  contains  the  major 
program  milestones  dated  from  start  of  contract  award,  major  program  activities,  and  their 
dependencies.  The  contractor  details  the  IMS  activities  to  accomplish  each 
event/accomplishment/criteria  entry  in  the  IMP.  The  IMS  is  updated  monthly.  At  a  minimum. 
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the  IMS  has  Work  Packages  for  the  next  12  months  and  Planning  Packages  thereafter.  A  Planning 
Package  consists  of  1  or  more  Work  Packages.  A  Work  Package  consists  of  one  or  more  detail 
tasks.  Ideally  a  detail  task  has  a  singularized  product,  and  defined  completion  criteria,  although 
this  is  not  always  true  in  practice. 

The  "Work  Package"  and  "Planning  Package"  identifiers  link  the  schedule  to  EVM  reporting. 
Reliability,  validity,  bias  and  accuracy  of  EVM  reporting  are  concerns,  as  is  the  resolution  of 
activity  decomposition,  EVM  reporting,  and  completion  criteria. 

For  each  activity,  the  IMS  contains  the  following  data  fields:  parent  WBS  element,  IMP  entry. 
Planning  or  Work  Package;  activity  scheduling  logical  dependency  relationships  (predecessor-to- 
successor  Finish-to-Start,  Start-to-Start,  Finish-to-Finish,  and  Start-to-Finish  dependencies); 
budgeted  time,  budgeted  cost,  actual  time  expended,  slack  time,  actual  cost  expended,  fraction 
of  work  performed.  Some  of  this  information  comes  from  the  EVM  system.  Level-of-effort  tasks 
are  not  included  in  the  IMS.  The  IMS  is  the  basis  for  schedule  and  cost  risk  analysis.  The  analysis 
can  be  for  the  overall  program,  by  WBS  element,  and/or  by  IMP  entry.  By  including  maturity 
advancement  steps  in  the  IMP,  cost  and  schedule  of  maturity  advancement  is  tracked. 

The  SSS  begins  with  the  performance  specification  (P-Spec),  a  part  of  the  RFP  package.  The  SSS 
contains  derived  requirements  for  system  segments  of  the  contract  WBS.  The  SSS  contains  the 
following  fields:  the  WBS  element  (level  3  or  below);  the  parent  SSS  elements  (there  may  be 
more  than  one);  the  singular  property  or  characteristic;  the  threshold  and  objective  criteria 
(performance  levels  and  conditions  of  performance),  priority  level  (e.g.  "tiers"),  pointers  to  the 
verification  tasks  in  the  IMS  (null  if  no  task  accomplishes  verification);  verification  results  (null  if 
the  verification  task  has  not  been  completed,  else  the  performance  measure  results  and  test 
conditions);  method  of  verification  (e.g.,  design  inspection,  analysis,  bench  testing,  integrated 
testing),  compliance  (compliant,  partially  compliant,  not  compliant);  estimate  of  achievable 
performance;  tradespace  dimension  and  tradeoffs  (holistic  system  attribute  tradespace  gained 
or  lost  if  the  requirements  are  changed  to  the  achievable  level;  e.g.,  size,  weight,  power,  cooling, 
production  cost,  and  impact  on  other  performance  requirements);  time  and  cost  of  verification 
if  there  is  not  a  verification  task.  Ideally,  the  SSS  addresses  the  costs  of  compliance  verification. 

Compliance  verification  can  be  at  different  levels,  e.g.,  design  acceptance,  manufacturing  and 
design  acceptance,  modeling  and  simulation  of  design  and  manufacturing,  system  integration 
laboratory  testing,  field  testing  etc. 

The  information  in  the  SSS  is  the  essential  information  for  tracking  compliance  progress,  and  for 
performance  requirements  trades  against  time,  cost,  and  design  tradespace.  It  provides 
information  for  time  and  cost  tradeoffs  versus  verification  to  reduce  risk.  It  tracks  back  to  the 
IMS  to  correlate  compliance  with  cost  and  schedule. 

The  SSS  would  benefit  from  a  prior  definition  of  the  holistic  attribute  tradespace  dimensions,  e.g., 
extracted  from  the  Ground  System  Architecture  Framework  [26],  and  linking  the  dimensions  to 
the  subsystems  that  supply  and  consume  each  resource.  It  would  also  benefit  from  a  formalized 
definition  of  the  levels  of  verification  matched  to  the  levels  of  the  maturity  level  definition,  e.g., 
verification  by  design  review,  subsystem  modeling  and  simulation,  subsystem  testing  in  a 
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laboratory  environment,  integrated  system  modeling  and  simulation,  integrated  system  testing 
in  representative  context,  full  system  integrated  testing  in  a  realistic  context  and  environment. 

The  ST  contains  the  technical  baseline  as  it  is  developed  during  the  program.  It  specifies  the 
functional  baseline,  allocated  baseline  and  product  baselines  [21].  Data  structures,  file  formats, 
data  fields  and  formats  have  not  been  standardized.  The  functional  baseline  describes  the 
functional  and  interface  characteristics  of  the  overall  system,  and  the  verification  required  to 
demonstrate  their  achievement.  The  allocated  baseline  defines  the  lower-level  configuration 
items  making  up  a  system,  and  how  system  function  and  performance  requirements  are  allocated 
across  lower  level  configuration  items,  including  design  constraints  and  the  verification  required 
to  demonstrate  the  traceability  and  achievement  of  specified  functional,  performance,  and 
interface  characteristics.  The  product  baseline  describes  the  functional  and  physical 
characteristics  of  a  configuration  item;  the  selected  functional  and  physical  characteristics 
designated  for  production  acceptance  testing;  and  tests  necessary  for  deployment/installation, 
operation,  support,  training,  and  disposal  of  the  configuration  item.  The  initial  product  baseline 
includes  "build-to"  specifications  for  hardware  (product,  process,  material  specifications, 
engineering  drawings,  and  other  related  data),  and  "code  to"  specifications  for  software. 
Verification  of  completeness  of  the  technical  baselines  is  normally  reviewed  at  technical  reviews, 
as  specified  in  the  IMP. 

The  SA  defines  the  physical  and  logical  system  entities  with  boundary  diagrams  and  behaviors. 
Boundary  diagrams  specify  entity  relationships  with  each  other  and  the  external  environment, 
and  are  supplemented  with  formatted  data  describing  the  characteristics  of  each  interface. 
Behaviors  are  derived  by  tracing  operational  scenarios,  vignettes,  mission  threads,  and/or  use 
cases  through  system  boundaries  to  derive  internal  system  behavior  culminating  with  a  linked 
and  traceable  allocation  of  behavior  for  product  design  and  development.  Specific  guidelines  for 
the  system  architecture  and  design  documentation  have  distribution  restrictions. 

System  architecture  assessment  can  include  technology,  integration,  and  manufacturing 
readiness  assessments.  These  assessments  are  for  selected  subsystems  and  technologies  for  CTE 
and  OTI  -  not  the  entire  system.  Reporting  TRL/IRL/MRL  for  all  system  architecture  elements 
and  levels  of  decomposition  would  support  risk  early  warning  across  the  entire  system  to  reveal 
evidence  of  previously  unforeseen  risks.  Restricting  TRL/IRL/MRL  assessment  to  CTEs  and  OTIs 
elevates  the  risk  of  being  blindsided  by  unforeseen  challenges  and  events.  However  detailed 
investigation  of  TRL/IRL/MRL  and  potential  risks  requires  commitment  of  time,  cost,  and  decision 
authority. 

The  TRC  are  filled  out  at  each  of  the  major  reviews,  based  evidence  presented  at  the  review  and 
judgment  of  the  reviewers.  The  TRC  probe  the  status  of  the  technical  program  in  thirteen  areas 
with  specific,  predetermined  questions  [28],  Status  is  rated  red,  amber,  green,  unknown,  or  not 
applicable.  The  TRC  cover  many  different  factors,  from  "was  a  responsible  person  appointed  to 
conduct  and  approve  the  review"  to  "what  fraction  of  the  critical  requirements  have  been  shown 
to  have  been  met." 

The  MCE  contains  estimates  of  the  recurring  and  non-recurring  costs,  for  each  variant  in  the 
Family  of  Vehicles,  against  a  detailed  standardized  Ground  System  Architecture.  The 
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Government  provides  the  reporting  framework.  The  contractor  provides  an  update  to  the 
estimate  at  each  major  program  review.  All  entries  are  initially  rated  "to  be  determined". 


Integrated  Performance  RLI 

The  Risk  Working  Group,  led  by  the  Army  Materiel  Systems  Analysis  Agency  (AMSAA)  identified 
lack  of  methods  for  integrated  cost,  schedule,  and  program  technical  performance  risk 
assessment  as  their  greatest  concern  and  highest  priority.  Methods  for  cost  and  schedule  risk 
analysis  were  not  coupled  with  assessment  and  diagnosis  of  what  the  risk  exposure  meant  for 
the  technical  performance  of  the  acquisition  program.  Technology  risk  assessment  methods  ask 
for  estimates  of  the  time  and  cost  consequences  if  identified  future  events  occur,  but  not  as  part 
of  integrated  cost  and  schedule  risk  assessment. 

In  principle,  cost  and  schedule  risk  analysis,  using  data  provided  in  IMS  and  EVM  reporting,  can 
be  used  to  analyze  time  and  cost  risk  exposure  by  technology  area,  stage  and  extent  of 
integration  and  incremental  verification  through  developmental  testing.  But  this  is  only  possible 
if  the  activities  in  the  IMS  are  mapped  to  design,  integration,  and  test  stages,  by  technology  area. 
The  mechanism  for  the  mapping  is  the  IMP.  The  Government  specifies  the  IMP  events, 
accomplishments  and  criteria  as  part  of  the  RFP  package.  The  IMP,  together  with  the  WBS,  are 
the  basis  for  the  IMS  and  EVM  reporting. 

While  the  mechanism  is  available,  the  IMP  has  nottraditionally  been  used  to  ensure  that  contract 
data  reporting  provides  the  information  needed  for  integrated  cost,  schedule  and  technical 
program  performance.  Events,  accomplishments,  and  criteria  in  the  IMP  typically  include  process 
steps  for  technical  reviews  (e.g.,  the  PRD  is  an  event,  PDR  preparation  completed  is  an 
accomplishment,  and  proceeding  to  PDR  approval  by  the  SEIT  is  a  criterion),  as  well  as  end-state 
events  such  as  prototype  deliveries  and  EMD  test. 

The  IMP  could  be  used  to  ensure  that  data  in  the  IMS  and  EVM  reports  are  suitable  to  assess  risk 
exposure  by  technical  area  and  advancement.  The  key  is  to  define  events,  accomplishments,  and 
criteria  that  correspond  to  significant  system  design  and  integration  stages  of  advancement.  We 
have  developed  two  complementary  methods.  Both  are  relevant.  They  provide  orthogonal 
views  of  program  technical  progress.  One  view  is  based  on  the  WBS,  the  other  is  based  on  the 
design,  manufacturing  and  RAM  maturity  levels  as  defined  in  the  RFP  attachments. 

Using  the  WBS  to  Define  Technical  Progress  Events 

The  WBS  is  an  attachment  to  the  RFP  package.  The  Government  defines  the  WBS.  The  WBS  is 
the  breakdown  structure  of  what  is  to  be  developed  and  delivered  under  the  contract.  The  WBS 
is  broken  down  by  functional  component,  e.g.,  automatic  ammunition  handling,  navigation  and 
remote  piloting,  etc.  The  WBS  also  includes  system  level  activities  such  as  integration,  assembly, 
test,  checkout,  shipping,  etc. 

Completion  of  each  functional  component  WBS  line  should  become  an  event  in  the  IMP.  The 
accomplishments  are  design,  integration,  developmental  testing,  manufacturing  development. 
The  system  level  activities  should  also  be  included  as  events. 
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Cost  and  schedule  reporting  tagged  to  these  events  and  accomplishments  enables  the  risk 
analysis  to  (1)  analyze  cost  and  schedule  risk  of  the  functional  components,  then  (2)  identify  the 
functional  components  contributing  most  to  overall  risk  exposure.  This  creates  the  ability  to 
analyze  risk  exposure  by  WBS  functional  component. 

Using  the  Maturity  Levels  to  Define  Technical  Progress  Events 

In  the  proposal  evaluation,  the  maturity  levels,  and  the  substantiating  data,  are  inputs  to  assess 
the  relative  risks  of  alternative  proposals.  As  the  program  advances  in  system  design, 
manufacturing  and  RAM  maturity,  risk  is  retired.  Maturity  advancement  ends  with  delivery  the 
completed  product.  The  maturity  level  definitions  can  be  used  integrate  time,  cost,  and  technical 
advancement  reporting  and  analysis  by  inserting  maturity  advancement  steps  into  the  IMP. 

Maturity  advancement  can  be  inserted  into  the  IMP  by  adding  three  events:  design  maturity 
advancement,  manufacturing  maturity  advancement,  and  RAM  maturity  advancement.  The 
following  notional  example  illustrates  how  the  maturity  advancement  could  be  resolved  into  IMP 
events,  accomplishments  and  criteria,  thus  forcing  traceable  linkage  between  technical  progress, 
time,  and  cost.  The  accomplishments  are  stages  of  maturity  advancement  towards  completion 
for  design,  manufacturing,  and  RAM. 

The  design  maturity  accomplishments  and  criteria  are  design  and  development  artifacts  and  test 
results,  corresponding  to  major  technical  reviews  [29]  and  test  &  evaluation  events  [30],  The 
manufacturing  maturity  accomplishments  and  criteria  are  drawn  from  the  manufacturing 
readiness  handbook  [31].  The  RAM  maturity  accomplishments  and  criteria  are  drawn  from  the 
design  for  reliability  handbook  [32], 

Event:  System  Design  Maturity  Advancement 

•  Accomplishment  1:  Requirements.  Criteria:  Completion  of  (a)  requirements  decomposition, 
(b)  functional  baseline,  (c)  derived  requirements  development,  and  (d)  interface 
identification 

•  Accomplishment  2:  Preliminary  Design.  Criteria:  Completion  of  (a)  allocated  baseline,  (b) 
component  analysis  and  selection,  (c)  CAD  models,  (d)  mass  properties  models,  (e)  load  plan 
models,  (f)  hull/structure/frame  finite  element  models,  (g)  powertrain  and  mobility  models, 
(h)  survivability  models,  and  (i)  interface  models 

•  Accomplishment  3:  Critical  Design.  Criteria:  Completed  sub-system  architecture,  design, 
integration  and  testing  in  the  an  integration  lab  for  product  WBS  Configuration  Items 

•  Accomplishment  4:  System  Integration  &  Test  Readiness.  Criteria:  Completed  sub-system 
integration,  testing,  analysis  of  deficiencies,  and  corrective  measures  in  an  operationally 
relevant  environment  for  product  WBS  Configuration  Items 

•  Accomplishment  5:  Prototype  Delivery  and  Operational  Testing.  Criteria:  (a)  delivery  of 
prototypes,  (b)  completion  of  Operation  Testing,  (c)  corrective  action  plans  for  residual 
deficiencies 

•  Accomplishment  6:  Production  Readiness  and  System  Verification.  Criteria:  Completion  of 
(a)  final  design,  (b)  Technical  Data  Package  with  CAD  models,  mass  properties  models,  loading 
plan/model,  and  bill  of  materials,  (c)  live  fire  test 

Event:  Manufacturing  Maturity  Advancement 
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•  Accomplishment  1:  Manufacturing  Concept.  Criteria:  Completed  (a)  initial  manufacturing 
and  process  models,  (b)  materials  acquisition  approach 

•  Accomplishment  2:  Manufacturing  Requirements  Analysis.  Criteria:  Completed 

identification  of  (a)  manufacturing  concepts,  (b)  producibility  needs,  (c)  new  manufacturing 
processes,  (d)  new  manufacturing  skills,  (c)  special  facility  requirements,  (f)  supply  chain 
requirements 

•  Accomplishments:  Preliminary  Manufacturing  Analysis.  Criteria:  Completed  identification 
of  (a)  manufacturing  modeling  and  simulation  approaches,  (b)  lead  times  for  materials,  (c) 
exotic  materials  (hazardous,  difficult  to  obtain  and/or  process),  and  (d)  supply  chain  model 
and  potential  sources 

•  Accomplishment  4:  Detailed  Manufacturing  Analysis.  Criteria:  Completed  (a)  modeling  and 
simulation  analysis  at  the  component  and  subsystem  levels  to  determine  constraints,  (b) 
assessment  of  issues,  performance  and  reliability  of  similar  full  production  processes,  (c) 
identification  of  skill  sets,  special  skills  training  and  certification,  (d)  selection  of  supply  chain 
sources 

•  Accomplishments:  Manufacturing  Feasibility  Verification.  Criteria:  Completed  verification 
of  (a)  the  manufacturing  processes  in  a  production  relevant  environment,  (b)  availability  of 
workforce  skills,  (c)  adequate  facilities  and/or  facility  development  plans,  (d)  long-lead  items 
identified,  (e)  obsolescence/disposal  issues  identified,  (f)  supply  chain  and  supplier 
agreements  in  place 

•  Accomplishment  6:  Full  Manufacturing  Verification.  Criteria:  Completed  (a)  demonstration 
of  manufacturing  processing  in  a  production  relevant  environment,  (b)  specification  of 
manufacturing  workforce  resource  requirements  (staffing  and  floor  managers),  (c) 
specification  of  facility  capabilities,  (d)  long-lead  item  procurement  plan,  (e)  obsolescence 
plan 

Event:  RAM  Maturity  Advancement 

•  Accomplishment  1:  Requirements  Analysis.  Criteria:  Completed  (a)  reliability  continuous 
improvement  plan  to  meet  reliability  and  maintainability  requirements,  (b)  reliability  (mean 
miles  between  system  abort)  and  maintainability  (maintenance  ratio,  mean  time  to  repair, 
max-time  to  repair)  allocated  down  to  the  Line  Replaceable  Unit  (LRU)  level 

•  Accomplishment  2:  Preliminary  Design  Analysis.  Criteria:  Completed  (a)  design  failure 
modes  and  effects  analysis  (DFMEA),  (b)  Fault  Tree  Analysis  (FTA)  for  all  essential  functions 
listed  in  the  Failure  Definition  and  Scoring  Criteria,  (c)  Critical  Items  List  -  items  whose  failure 
would  cause  a  mission  failure  or  Category  III  or  higher  Hazard  Severity  Rating  as  defined  in 
MIL-STD-882D,  (d)  reliability  and  maintainability  estimates  made  at  the  LRU  level,  (e)  initial 
reliability  growth  plan  and  curve  (per  AMSAA  Projection  Maturity  Model) 

•  Accomplishment  3:  Integrated  Subsystem  Reliability.  Criteria:  Completed  modeling  and 
simulation  and/or  sub-system  testing  to  estimate  the  reliability  of  integrated  sub-system 
operation 

•  Accomplishment  4:  Detailed  Design  Analysis.  Criteria:  Using  integrated  subsystem  reliability 
date  from  Accomplishment  3,  completed  (a)  updated  DFMEA,  (b)  updated  FTA,  (c)  updated 
reliability  and  maintainability  estimates,  (d)  reliability  growth  plan  and  curve,  and  (e)  Failure 
Reporting  and  Corrective  Action  System  (FRACAS)  report 
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•  Accomplishment  5:  System  Level  Testing.  Criteria:  Completed  (a)  reliability,  maintainability, 
and  durability  test  plan,  (b)  testing  the  specified  number  of  miles  on  the  operational  terrain 
profile,  (c)  test  report  with  details  of  the  driving  profile,  terrain  profile,  times  and  types  of 
failures 

•  Accomplishment  6:  Post  Testing  Detailed  Design  Analysis.  Criteria:  Using  system  level  testing 
data  from  Accomplishment  5,  completed  (a)  updated  FRACAS  report  identifying  failure 
modes,  root  causes,  corrective  actions,  and  validation,  (b)  updated  reliability  and 
maintainability  predictions,  and  (c)  updated  reliability  growth  plan  and  curve 


Risk  Leading  Indicators 

Risk  Leading  Indicators  are  computed  from  baseline  and  update  program  management  and 
systems  engineering  data.  RLI  assess  the  current  status,  as  well  as  trends  and  instability.  They 
assess  (1)  the  change  from  the  initial  baseline,  and  (2)  the  change  from  the  last  update.  This 
requires  that  the  Systems  Engineering  system  retain  initial  baselines  and  previous  state.  This  is 
the  minimum  information  to  assess  current  state,  short-  and  long-term  trends.  The  reporting 
process  samples  the  state  of  the  system  in  periodic  updates  (e.g.,  monthly)  and  event-based 
updates  (e.g.,  technical  reviews).  Early  warning  of  risk  exposure  and  emerging  risk  exposure 
requires  early  visibility  into  the  state  of  the  system  and  its  progress.  Waiting  until  late-stage 
technical  reviews  to  identify  issues  exposes  the  program  to  risk. 

The  initial  list  of  candidate  Risk  Leading  Indicators  follows.  The  RLI  address  requirements, 
maturity,  design  (baselines,  architecture,  and  design  margins),  technical  reviews,  plan  and 
schedule. 

•  Requirements  Interdependency.  Number  of  requirements,  number  of  dependencies 
between  requirements,  ratio 

•  Requirements  Stability.  Proportion  of  requirements  and  dependencies  that  were  (a)  added, 
(b)  deleted,  and  (c)  changed 

•  Requirements  Verification.  Proportion  of  requirements  with  compliance  verification 
activities  in  the  IMS,  number  that  have  been  verified,  number  that  were  fully/partially/not 
compliant,  number  of  requirements  linked  to  requirements  that  were  fully/partially/not 
compliant 

•  Requirements  Verification  Schedule.  Minimum  and  average  schedule  slack  for  remaining 
verification  activities,  minimum  and  average  project  time  remaining  after  scheduled 
verification  activities 

•  Maturity.  Design,  manufacturing  and  RAM  maturity  levels,  difference  between  scheduled 
level  and  achieved  level  (at  the  criteria,  accomplishment,  and  event  levels) 

•  Manufacturing  Cost  Maturity.  Proportion  of  applicable  entries  in  the  MCE  that  are  blank  or 
"to  be  provided",  the  proportion  of  entries  with  different  values  than  the  previous  submittal; 
percentage  change  in  the  estimated  unit  production  cost 

•  Baselines.  Number  of  entries  in  the  functional,  allocated,  and  product  baselines,  number  of 
links  to  requirements,  number  of  links  to  architecture  elements,  ratios  of  changes  to  number 
in  each  baseline 
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•  Architecture.  Numbers  of  architecture  elements  and  links  between  elements,  proportion 
without  completed  boundary  diagrams,  proportion  with  changed  boundary  diagrams 

•  Design  Margins.  Design  margin  remaining  as  a  proportion  of  the  base,  for  each  holistic  system 
parameter  (ground  vehicle  holistic  system  parameters  are  found  in  the  Ground  System 
Architecture  Framework  [26],  e.g.,  mass  properties,  volume,  dimensions,  surface  areas,  main 
and  auxiliary  power,  cooling  capacity,  fuel  consumption,  etc.) 

•  Technical  Reviews.  Total  and  by  major  sub-heading  the  number  of  applicable  fields,  for  the 
applicable  fields,  the  proportion  rated  red,  amber,  green,  and  unknown,  number  of  changes 
from  unknown  to  known,  proportions  of  increases  and  decreases  in  level  and  from  unknown 
to  known  and  known  to  unknown 

•  Integrated  Master  Plan  (IMP)  Stability.  Number  of  IMP  entries,  proportion  added,  deleted, 
or  changed 

•  Integrated  Master  Schedule  (IMS).  Number  of  IMS  activities  (by  Planning  Package,  Work 
Package,  and  detail  task),  number  of  dependencies  (by  logical  type),  ratio  of  links  to  nodes, 
proportion  of  activities  completed,  begun,  started  or  finished  out  of  sequence  with  their 
dependencies,  mean  and  variance  of  the  ratio  of  actual  to  planned  duration  for  completed 
activities,  float  on  the  critical  path  relative  to  the  critical  path  to  each  milestone,  number  of 
"high  risk"  activities  (i.e.,  with  ratio  of  float  to  scheduled  duration  is  less  than  X  percent) 

•  Schedule  Stability.  Numbers  and  proportions  of  activities  and  dependencies  that  were  (a) 
added,  (b)  deleted,  and  (c)  changed 


Areas  of  High  Relative  Risk 

Risk  Leading  Indicators  can  be  computed  for  the  entire  program  and  system  to  assess  overall  risk 
exposure,  and  by  segment  to  diagnose  areas  of  high  risk  exposure.  Different  RU  are  decomposed 
in  different  ways  that  give  different  insight  into  the  areas  at  risk.  Requirements  RU  are  linked  to 
priority  tier,  and  branch  of  the  specification  tree.  Maturity  RU  are  linked  to  acquisition  phase 
(development,  manufacturing,  and  sustainment)  and  major  subsystem.  Design  RU  are  organized 
by  product  WBS.  Schedule  RU  are  linked  to  WBS  element,  and  by  IMP  entry.  Technical  review 
checklists  have  their  own  organization. 

Risk  Leading  Indicators  that  are  correlated  can  be  combined  using  Principal  Components  Analysis 
to  reduce  the  number  of  indicators  and  variability.  This  requires  an  accumulation  of  data  points. 

Predicting  future  time,  cost,  performance  differences  between  actual  and  planned  results 
(assuming  that  past  relationships  between  RLI  and  subsequent  outcomes  will  persist)  requires  an 
accumulation  of  data  points  by  IMS  activity. 

Some  decisions  need  to  be  made  before  there  has  been  enough  history  to  compute  trend  and 
stability  metrics,  and  before  there  has  been  time  to  accumulate  data  to  calibrate  statistical  RERs. 
"Shortcut"  methods  are  needed  to  assess  relative  risk  (a)  in  proposal  evaluation  without  "trend 
and  stability"  data,  (b)  during  contract  execution  but  without  sufficient  history  to  quantify  time, 
cost,  and  performance  bias  and  dispersion,  and  (c)  when  there  is  insufficient  data  to  project  bias 
and  dispersion  of  outcome. 
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The  underlying  concept  is  to  find  outliers,  e.g.,  WBS  elements  with  high  levels  of  RU  relative  to 
other  WBS  elements.  Even  without  data  to  quantify  the  relationship  between  RU  and  time,  cost, 
performance  outcome,  it  is  still  possible  to  identify  WBS  elements  with  elevated  RU  -  e.g.  if  one 
proposal  or  WBS  element  has  RU  that  are  three  standard  deviations  above  the  norm,  it  is  higher 
risk  than  one  whose  RU  are  one  standard  deviation  above  the  norm. 


Risk  Estimating  Relationships  (RERs) 

RERs  are  equations  or  models.  The  RU  are  the  inputs.  Time,  cost  and  performance  bias  and 
uncertainty  are  outputs.  RERs  are  calibrated  to  program  data  on  time,  cost,  and  performance  by 
IMP  entry,  WBS  element,  and  the  overall  program.  Calibration  uses  historical  data  on  the 
program  to  compute  model  coefficients.  Differences  between  IMP  entry,  WBS  element,  and 
entire  program  provide  insight  into  the  sources  of  risk  exposure. 

Risk  Estimating  Relationships  are  essentially  regression  models  that  explain  future  IMS  activity 
cost  and  schedule  bias  and  dispersion  in  terms  of  current  risk  leading  indicators.  RERs  are  similar 
to  Cost  Estimating  Relationships  (CERs)  used  in  parametric,  analogy  cost  models.  RERs  are 
statistically  significant  relationships  that  explain  program  performance  as  functions  of  earlier  RU. 

The  RER  are  calibrated  to  previous  period  data  for  the  current  program.  Program-to-program 
differences  make  it  unlikely  that  quantitative  evidence  from  one  program  will  be  relevant  to 
another  program  with  different  design  challenges,  performance  objectives,  contractor 
management,  engineering  team,  etc.  The  RER  contain  autoregressive  components  (past  trends 
predict  future  trends)  and  logical  components  (incompleteness,  instability,  inconsistency,  lack  of 
safety  margins,  interdependency  in  the  current  state  predict  deficiencies  in  future  outcomes). 

Program  performance  data  are  accumulated  over  time.  At  proposal  evaluation,  only 
uncertainties  inherent  in  the  RFP  package  and  the  proposals  can  be  assessed.  After  contract 
award,  data  can  be  accumulated  regarding  input  conditions  and  output  results,  by  IMP  entry, 
WBS,  and  overall  program. 

Each  completed  activity  in  the  IMS  constitutes  a  data  point  with  a  cost  and  schedule  variance. 
Completed  requirements  verification  tasks  also  provide  compliance  variance.  Each  IMP  event  is 
a  data  point. 

Regression  models  include  all  computational  methods  that  use  historical  data  to  fit  a  type  of 
model  between  input  and  output  variables.  Candidate  regression  methods  include  linear,  multi¬ 
linear,  non-linear,  locally  linear,  and  adaptive  statistical  regression.  Other  statistical  estimation 
frameworks  include  naive  Bayes  models  and  families  of  multi-dependency  Bayes  models  with 
fixed  or  adaptive  weighting. 

The  choice  of  underlying  RER  models  is  pragmatic  -  the  formalism  that  works  best  is  best  to 
explain  progress  variances.  Previous  programs  can  suggest  high  value  models  and  parameters, 
but  differences  among  program  may  reduce  relevance. 

The  question  is  which  methods  work  -  in  practice,  for  this  application  -  not  which  are  best  in 
theory.  The  characteristics  of  the  application  are  complex.  Different  programs  will  have  different 
challenges.  This  makes  extrapolating  from  on  program  to  another  problematic.  There  are  a  large 
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number  of  potential  risk  exposure  indicators.  In  any  given  program,  at  any  stage  of  the  program 
(punctuated  by  the  major  technical  reviews),  and  divided  by  the  WBS  elements,  different 
indicators  can  have  different  significance.  Outcome  states  can  be  positively  or  negatively 
correlated.  EMD  time  and  cost  tend  to  be  positively  correlated,  and  negatively  correlated  with 
system  performance,  production  cost  and  RAM. 

The  large  number  of  IMS  activities  provide  data  to  correlate  WBS/IMP/IMS  time/cost  expenditure 
and  technical  accomplishment  to  the  RU  (e.g.,  IMS  limited  to  6,000  detail  tasks).  The  semi- 
hierarchical  lattice  structure  of  requirements,  program  activities,  system  segments,  and  system 
architecture  pose  challenges  to  statistical  analysis,  as  does  potential  differences  in  the  program 
between  technical  review  milestones  and  level  3  WBS  elements. 
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Future  Objectives  and  Approach 


The  primary  objective  for  2015  is  to  "stress-test"  the  RU  via  pilot  application  to  an  EMD-phase 
MDAP,  and  potentially  to  a  pre-EMD  phase  program.  Different  RU  apply  at  pre-EMD  and  EMD 
stages. 

The  pilot  applications  will  be  used  to  refine  &  adapt  the  RU,  to  demonstrate  and  assess  their 
value  to  the  program,  and  to  identify  potential  issues  that  might  impede  transition  into  general 
use.  The  pilot  applications  will  be  conducted  in  collaboration  with  the  programs'  Risk  Managers, 
and  the  TARDEC  Risk  Management  team.  We  met  with  the  MDAP  Risk  Manager  and  TARDEC  risk 
management  team  in  early  February  2015,  to  review  and  refresh  our  collaboration  agreement 
following  contract  award  and  exit  from  Source  Selection  in  late  December  2014.  Specifics  of  the 
plans  to  pilot  the  EMD-phase  RU  on  the  MDAP  are  described  in  the  following  section. 

We  have  a  meeting  scheduled  in  late  February  2015  to  meet  with  the  pre-EMD  "Future  Tank" 
team  (PM  Abrams)  to  discuss  collaboration.  The  "Future  Tank"  program  is  a  significant  upgrade 
to  the  Abrams  tank,  with  increased  modularity  for  potential  to  integrate  advanced  technologies 
as  they  mature.  This  report  is  being  prepared  prior  to  the  meeting.  There  is  potential  to  open 
discussions  with  the  CVP  (Combat  Vehicle  Prototype)  technology  demonstration  project  at 
TARDEC,  although  we  have  no  plans  to  do  so  at  this  time.  The  CVP  is  an  evolution  of  the  Ground 
Combat  Vehicle  (GCV),  pushed  back  to  a  Technology  Demonstration  project  vice  a  MDAP. 

A  secondary  objective  is  to  communicate  the  RU  MPT  and  the  pilot  application  emerging  results 
to  the  broader  user  and  developer  community.  The  will  be  accomplished  through  a  series  of 
discussions  and  presentations,  culminating  in  a  workshop  at  the  end  of  the  year.  The  intended 
audience  of  the  workshop  includes  end-users  such  as  AMSAA  and  Risk  Management  Teams  from 
RDECOM  elements  outside  of  TARDEC,  and  other  service  components,  as  well  as  developers  and 
theoreticians  from  groups  such  as  RAND,  IDA,  and  MITRE.  This  will  be  a  continuation  and 
extension  of  the  technical  exchanges  held  during  2014. 

A  third  objective  is  to  begin  to  develop  RU  that  are  specific  development  risks  associated  with 
software-intensive  cyber-physical  systems  with  real-time,  distributed,  embedded  processing 
interacting  with  sensor,  actuator,  communications  hardware,  and  the  external  system  dynamics. 
This  will  be  accomplished  by  building  on  RU  findings  in  2014,  related  research  on  other  research 
tasks,  and  coordinating  with  DASD-SE  personnel  with  related  objectives  and  responsibilities. 

Preliminary  findings  indicate  that  the  number  and  density  of  interconnections  and 
interdependencies  among  the  cyber-physical  components  is  a  driving  factor  in  development  time 
and  cost,  and  potential  needed  for  rework  to  correct  for  unforeseen  omissions  and  interactions 
[33,  34,  35],  The  experience  of  the  technical  management  and  engineering  leads  with  similar 
systems  is  another  important  factor  in  time  and  cost  estimating  accuracy  and  uncertainty.  The 
challenge  is  to  develop  risk  leading  indicators  is  to  forecast  the  density  of  interconnections  and 
interdependencies  of  the  eventual  system  design  during  earlier  the  conceptualization  and 
requirements  formulation  stages. 
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EMD-Phase  RLI  Pilot  Application  Plans 


2014  we  established  a  collaboration  agreement  with  an  MDAP  (under  PEO  GCS)  to  pilot  the  RLI 
on  the  program,  provided 

1.  that  we  would  coordinate  with  and  provide  feedback  to  the  risk  manager  so  that  the 
program  would  receive  the  benefits 

2.  that  we  would  work  with  data  and  reports  that  the  program  could  provide  with  minimal 
additional  cost  or  effort 

3.  that  the  support  to  the  program  would  be  sustained  collaboration  vice  a  "one-off"  study, 
and 

4.  that  we  collaborate  with  TARDEC  so  that  the  finding  and  results  would  transition  to  the 
end-user  community. 

At  the  time  of  the  agreement,  the  program  was  about  to  enter  Source  Selection,  at  which  point 
communication  with  the  PMO  was  to  be  suspended  in  order  to  prevent  possible  legal 
complications.  The  contract  was  awarded  in  late  December  2014,  with  a  start-of-work  meeting 
in  January  2015.  During  this  time,  the  PMO  was  reorganized  and  a  new  Risk  Manager  was 
assigned.  We  held  an  "re-start"  meeting  with  the  Risk  Manager  and  TARDEC  Risk  Management 
team  in  early  February  2014,  with  a  collaboration  launch  meeting  planned  for  early  March  2015. 
The  AMPV  PDR  is  scheduled  for  June,  2015,  with  CDR  in  summer,  2016. 

The  current  plan  is  to  brief  the  PM  or  his  deputy,  and  the  Risk  Manager,  in  March  2015  on  (1) 
risks  embedded  in  the  RFP  package  and  how  they  can  be  mitigated,  and  (2)  plans  to  support  the 
Risk  Manager  via  piloting  the  RLI.  The  briefing  will  define  specific  contract  data  reporting  items 
we  request,  and  explain  how  they  will  be  used  to  evaluate  which  specific  RLI,  and  the  significance 
of  the  RLI.  The  briefing  will  also  explain  the  collaboration  plan.  The  current  collaboration  plan  is 
to  receive  monthly  updates  to  the  CDRI  from  the  PMO,  and  to  provide  quarterly  briefings  to  the 
Risk  Manager.  In  addition  to  regular  quarterly  briefings,  we  will  participate  in  risk  management 
meetings,  as  needed  to  examine  specific  risk  exposure  evidence  and  potential  mitigation. 

Contract  Data  Reporting  Items 

Only  unclassified,  non-proprietary,  contract  data  reporting  items  will  be  used,  whose  distribution 
control  statement  allows  release  to  Government  contractors.  Initial  releases  and  their  periodic 
updates  are  needed  so  that  the  RLI  will  reflect  the  current  status  and  trends  in  the  program.  The 
data  items  of  interest,  as  currently  identified,  are: 

•  Contractor  Work  Breakdown  Structure  (CWBS)  -  an  elaboration  of  the  Government  WBS 

•  Contractor  Systems  Engineering  Management  Plan  (SEMP  -  which  documenting  the 
Integrated  Product  Teams  and  team  experience  and  other  important  Systems  Engineering 
factors) 

•  Contractor  Integrated  Master  Plan  (IMP)  -  an  elaboration  of  the  Government  IMP 

•  Contractor  Integrated  Master  Schedule  (IMS)  -  documenting  the  planned  chains  of  events 
and  activities,  durations,  and  current  status  of  the  program,  linked  to  the  WBS  and  IMP 
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•  Earned  Value  Management  (EVM)  reports  on  the  budgeted  cost  of  work  performed,  actual 
cost  of  work  performed,  and  proportion  of  work  performed  at  the  "work  package"  level  of 
the  IMS 

•  Selected  proposal  and  system  description  sections  -  as  referenced  in  the  "Matrix  of 
Substantiating  Data  for  System  Design  Maturity,  Manufacturing,  and  RAM"  (RFP  attachment 
0052) 

•  Manufacturing  cost  estimates  (per  the  template  in  RFP  attachment  0067) 

•  Risk  register  (list  and  status  of  currently  identified  risks,  per  the  Risk  Management  Plan,  RFP 
attachment  002) 

•  System  Architecture  Description  Document  (SADD,  per  RFP  attachment  0104,  Integrated 
System  Architecture  Modeling  Guide,  RFP  attachment  0087,  and  Ground  System  Architecture 
Framework,  RFP  attachment  0030) 

•  Technical  review  checklists  when  completed  following  technical  reviews 

High  Value  and  Second  Tier  RLI 

In  discussions  with  the  Risk  Manager  several  areas  of  concern  and  applicable  RLI  were  discussed. 

Integrated  cost,  schedule  and  technical  progress  risk  was  a  major  concern.  Program  cost  and 
schedule  risk  analysis  were  not  linked  to  measures  of  technical  progress  (i.e.,  maturity  level 
advancement)  and  technical  areas  (i.e.,  WBS  elements).  In  order  to  apply  the  RLI  and  RER 
formulated  to  address  this  issue,  we  need  to  map  the  IMS  activities  to  the  maturity  levels  and 
WBS  elements.  These  links  need  to  be  built.  Important  RLI  are 

1.  the  cost  and  schedule  bias  and  uncertainty  by  maturity  level  and  WBS  element 

2.  the  risk  to  overall  project  cost  and  schedule,  by  maturity  level  and  WBS  element 

The  Risk  Manager  acknowledged  that  the  schedule  risk  assessment  as  described  in  the  RFP 
package  was  limited,  and  did  not  provide  adequate  insight  into  schedule  risk.  Schedule  risk 
analysis  is  needed  to  addresses  high  risk  paths  (not  just  the  critical  path  and  the  impact  of 
uncertainty  on  the  critical  path),  and  identify  IMS  activities  with  a  large  role  in  overall  schedule 
risk.  The  schedule  RLI  are  appropriate  to  provide  this  additional  insight  and  early  warning. 

Second  tier  RLI  address  completeness  and  stability  of  various  plans  and  estimates,  specifically  the 
manufacturing  cost  estimate,  the  system  architecture,  and  the  IPT  composition.  The 
completeness  of  IPTs  matching  the  convergence  of  parallel  paths  in  the  IMS  was  also  discussed 
as  evidence  of  presence  of  risk  mitigation  via  the  IPTs. 

As  a  point  of  interest,  we  also  discussed  the  potential  value  of  examining  the  correlation  between 
the  RLI  and  (a)  risks  identified  through  the  standard  risk  management  process,  and  (b)  technical 
review  checklists  when  completed. 
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2015  Schedule  of  Major  Events 


February  2015 

•  Deliver  final  report  on  2014 

•  Reconnection  meeting  with  AMPV  Risk  Manager 

•  Initial  meeting  with  PM  Abrams  "Future  Tank"  team 

March  2015 

•  Phase  III  Kickoff  Meeting  or  Teleconference  with  DASD-SE 

•  Brief  the  MDAP  PMO  on  collaboration  plans,  risks  embedded  in  the  RFP,  and  mitigations 

•  Initiate  piloting  EMD-phase  RU  on  the  MDAP  as  source  material  is  received 

•  Initiate  piloting  pre-EMD  RU  on  "Future  Tank"  as  source  material  is  received 

April  2015 

•  Teleconference  with  Sean  Brady  DASD-SE,  MPS  on  risk  indicators  for  software-intensive 
cyber-physical  systems  with  real-time  embedded,  distributed  processing  coupled  to 
sensor,  actuator  and  communications  hardware 

May  2015 

•  Quarterly  meeting  with  the  MDAP  Risk  Manager  and  TARDEC  Risk  Management  liaison 

•  Quarterly  meeting  with  "Future  Tank"  and  TARDEC  Risk  Management  liaison 

•  Quarterly  teleconference  with  DASD-SE,  Major  Program  Support 

August  2015 

•  Quarterly  meeting  with  the  MDAP  Risk  Manager  and  TARDEC  Risk  Management  liaison 

•  Quarterly  meeting  with  "Future  Tank"  and  TARDEC  Risk  Management  liaison 

•  Quarterly  teleconference  with  DASD-SE,  Major  Program  Support 

September  2015 

•  Teleconference  with  Sean  Brady,  DASD-SE,  MPS  on  risk  indicators  for  software-intensive 
cyber-physical  systems  with  real-time  embedded,  distributed  processing  coupled  to 
sensor,  actuator  and  communications  hardware 

November  2015 

•  Quarterly  meeting  with  the  MDAP  Risk  Manager  and  TARDEC  Risk  Management  liaison 

•  Quarterly  meeting  with  "Future  Tank"  and  TARDEC  Risk  Management  liaison 

•  Quarterly  teleconference  with  DASD-SE,  Major  Program  Support 

December  2015 

•  Workshop  on  Risk  Leading  Indicators  inviting  AMSAA,  et.  al. 

•  Annual  meeting  with  and  presentation  to  DASD-SE,  MPS 

February  2016 
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•  Deliver  final  report  on  2015 

•  Quarterly  meeting  with  AMPV  Risk  Manager  and  TARDEC  Risk  Management  liaison 

•  Quarterly  meeting  with  "Future  Tank"  and  TARDEC  Risk  Management  liaison 
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